There has been an apparatus which satisfies the sense of real existence that a counterpart is present in front of you and aims at realistic picture communication by extracting, for example, a person's picture out of the scenery picture of a space in which you are present and superimposing the person's picture, a person's picture sent from the counterpart, and the picture of a virtual space to be displayed commonly with a previously-stored counterpart on each other and displaying them (Japanese Patent Publication No. 4-24914).
Particularly, in the case of the prior art, inventions concerned with acceleration for performing picture synthesis and a method for reducing memories are made (e.g. Official gazette of Japanese Patent Publication No. 5-46592: Picture synthesizer).
Though a communication system using picture synthesis for synthesizing two-dimensional static pictures or three-dimensional CG data has been proposed by the prior art, specific discussion on a method for realizing a system for simultaneously synthesizing a plurality of video(picture) and a plurality of audio and displaying them has not been performed from the following viewpoints.
That is, there has been a problem that no specific discussion has been performed from the following viewpoints:
(A1) a method for transmitting (communicating and broadcasting) and controlling pictures and audio under the environment in which data and control information (information transmitted by a packet different from that of data to control the processing of terminal side) are independently transmitted by using a plurality of logical transmission lines constructed by software on one real transmission line or more;
(A2) a method for dynamically changing header information (corresponding to data control information of the present invention) to be added to data for a picture or audio to be transmitted;
(A3) a method for dynamically changing header information (corresponding to transmission control information of the present invention) to be added for transmission;
(A4) a method for transmitting information by dynamically multiplexing and separating a plurality of logical transmission lines;
(A5) a method for transmitting pictures and audio considering the read and rise periods of program or data; and
(A6) a method for transmitting pictures and audio considering zapping.
However, the method for changing encoding systems and a method of discussing data in frames in accordance with the frame type of a picture have been proposed so far as a method for dynamically adjusting the amount of data to be transmitted to a network (H. Jinzenji and T. Tajiri, A study of distributive-adaptive-type VOD system, D-81, System Society of Institute of Electronics, Information and Communication Engineers (IEICE) (1995)).
A dynamic throughput scalable algorithm capable of providing a high-quality video under a restricted processing time is proposed as a method for adjusting throughput at the encoder side (T. Osako, Y. Yajima, H. Kodera, H. Watanabe, K. Shimamura: Encoding of software video using a dynamic throughput scalable algorithm, Thesis Journal of IEICE, D-2, Vol. 80-D-2, No. 2, pp. 444-458 (1997)).
Moreover, there is an MPEG1/MPEG2 system as an example of realizing synchronous reproduction of video and audio.
(B1) The conventional method for discussing a picture correspondingly to the frame type of the video has a problem that it is difficult to preponderantly reproduce an important scene cut synchronously with audio by handling a plurality of video streams or a plurality of audio streams and reflecting the intention of an editor because the grading of the information which can be handled is in a single stream.
(B2) Moreover, it must be possible that a decoder decodes every supplied bit stream because it is a prerequisite that MPEG1/MPEG2 is realized by hardware. Therefore, it is a problem how to correspond to the case of exceeding the throughput of the decoder.
Moreover, to transmit video, there have been some systems including a system such as H.261 (ITU-T Recommendation H.261-Video codec for audio-visual services at px 64) and they have been mounted by hardware. Therefore, the case has not occurred that decoding is not completed within a designated time because of considering the upper limit of a necessary performance when designing hardware.
The above-designated time denotes a time required to transmit a bit stream obtained by coding a sheet of video. If decoding is not completed within the time, an extra time becomes a delay. If the delay is accumulated, the delay from the transmitting side to the receiving side increases and the system cannot be used as a video telephone. This state must be avoided.
Moreover, when decoding cannot be completed within a designated time because a communication counterpart generates an out-of-spec bit stream, a problem occurs that a video cannot be transmitted.
The above problem occurs not only for a video but also for audio data.
However, in recent years, because the network environment formed by personal computers (PCs) has been arranged as the result of spread of internet and ISDN, the transmission rate has been improved and it has been possible to transmit a video by using PCs and a network. Moreover, requests for transmission of video by users have been rapidly increased. Furthermore, a video can be completely decoded by software because CPU performances have been improved.
However, because the same software can be executed by personal computers different in structure such as a CPU, bus width, or accelerator, it is difficult to previously consider the upper limit of a necessary performance and therefore, a problem occurs that a picture cannot be decoded within a designated time.
Moreover, when coded data for a video having a length exceeding the throughput of a receiver is transmitted, coding cannot be completed within a designated time.
Problem (C1): Decreasing a delay by decoding a picture within a designated time.
When inputting a video as the waveform data of the present invention or outputting a video as the waveform data of the present invention as means for solving the problem 1, a problem may be left that the substantial working efficiency of a transmission line is lowered because a part of a transmitted bit stream is not used. Moreover, there are some coding systems that generate a present decoded video in accordance with a last decoded picture (e.g. P picture). However, because the last decoded picture is not completely restored by the means for solving the problem 1, there is a problem that deterioration of the picture quality influentially increases as time passes.
Problem (C2): In the case of the means for solving the problem 1, the substantial working efficiency of a transmission line is lowered. Moreover, picture-quality deterioration is spread.
Furthermore, in the case of mounting by software, the frame rate of a picture is determined by the time required for one-time coding. Therefore, when the frame rate designated by a user exceeds the throughput of a computer, it is impossible to correspond to the designation.
Problem (C3): When the frame rate designated by a user exceeds the throughput of a computer, it is impossible to correspond to the designation.